SAMPLE GENERALIZED EIGENVALUE BASED DETECTION 
12 



SAMPLE GENERALIZED EIGENVALUE BASED DETECTION 2 

Fundamental limit of sample generalized eigenvalue 
based detection of signals in noise using relatively 
few signal-bearing and noise-only samples 

Raj Rao Nadakuditi and Jack W. Silverstein 
Abstract 

The detection problem in statistical signal processing can be succinctly formulated: Given m (possibly) signal 
bearing, n-dimensional signal-plus-noise snapshot vectors (samples) and N statistically independent n-dimensional 
noise-only snapshot vectors, can one reliably infer the presence of a signal? This problem arises in the context of 
applications as diverse as radar, sonar, wireless communications, bioinformatics, and machine learning and is the 
critical first step in the subsequent signal parameter estimation phase. 

The signal detection problem can be naturally posed in terms of the sample generalized eigenvalues. The sample 
generalized eigenvalues correspond to the eigenvalues of the matrix formed by "whitening" the signal-plus-noise 
sample covariance matrix with the noise-only sample covariance matrix. In this article we prove a fundamental 
asymptotic limit of sample generalized eigenvalue based detection of signals in arbitrarily colored noise when there 
are relatively few signal bearing and noise-only samples. 

Specifically, we show why when the (eigen) signal-to-noise ratio (SNR) is below a critical value, that is a simple 
function of n, m and N , then reliable signal detection, in an asymptotic sense, is not possible. If, however, the 
eigen-SNR is above this critical value then a simple, new random matrix theory based algorithm, which we present 
here, will reliably detect the signal even at SNR's close to the critical value. Numerical simulations highlight the 
accuracy of our analytical prediction and permit us to extend our heuristic definition of the effective number of 
identifiable signals in colored noise. We discuss implications of our result for the detection of weak and/or closely 
spaced signals in sensor array processing, abrupt change detection in sensor networks, and clustering methodologies 
in machine learning. 

Index Terms 

signal detection, random matrices, sample covariance matrix, Wishart distribution, multivariate F distribution 
EDICS Category: SSP-DETC Detection; SAM-SDET Source detection 

I. Introduction 

The observation vector, in many signal processing applications, can be modelled as a superposition of a finite 
number of signals embedded in additive noise. The model order selection problem of inferring the number of signals 
present is the critical first step in the subsequent signal parameter estimation problem. We consider the class of 
estimators that determine the model order, i.e., the number of signals, in colored noise from the sample generalized 
eigenvalues of the signal-plus-noise sample covariance matrix and the noise-only sample covariance matrix pair. The 
sample generalized eigenvalues [I] precisely correspond to the eigenvalues of the matrix formed by "whitening" the 
signal-plus-noise sample covariance matrix with the noise-only sample covariance matrix (assuming that the number 
of noise-only samples is greater than the dimensionality of the system so that the noise-only sample covariance 
matrix is invertible). 

Such estimators are used in settings where it is possible to find a portion of the data that contains only noise fields 
and does not contain any signal information. This is a realistic assumption for many practical applications such 
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as evoked neuromagnetic experiments [2]-[4], geophysical experiments that employ a "thumper" or in underwater 
experiments with a wideband acoustic signal transducer where such a portion can be found in a data portion taken 
before a stimulus is applied. In applications such as radar or sonar where the signals of interest are narrowband 
and located in a known frequency band, snapshot vectors collected at a frequency just outside this band can be 
justified as having the same noise covariance characteristics assuming that we are in the stationary-process-long- 
observation-time (SPLOT) regime [5]. 

Our main objective in this paper is to shed new light on this age old problem of detecting signal in noise from 
finite samples using the sample eigenvalues alone [6], [7]. We bring into sharp focus a fundamental statistical limit 
that explains precisely when and why, in high-dimensional, sample size limited settings underestimation of the model 
order is unavoidable. This is in contrast to works in the literature that use simulations, as in [8], to highlight the 
chronically reported symptom of model order estimators underestimating the number of signals without providing 
insight into whether a fundamental limit of detection is being encountered. 

In recent work [9], we examined this problem in the white noise scenario. The main contribution of this paper is 
the extension of the underlying idea to the arbitrary (or colored) noise scenario. Analogous to the definition in [9], 
we define the effective number of identifiable signals in colored noise as the number of the generalized eigenvalues 
of the population (true) signal-plus-noise covariance matrix and noise-only covariance matrix pair that are greater 
than a (deterministic) threshold that is a simple function of the number of signal-plus-noise samples, noise-only 
samples and the dimensionality of the system. Analogous to the white noise case, increasing the dimensionality of 
the system, by say adding more sensors, raises the detectability threshold so that the effective number of identifiable 
signals might actually decrease. 

An additional contribution of this paper is the development of a simple, new, algorithm for estimating the number 
of signals based on the recent work of Johnstone [10]. Numerical results are used to illustrate the performance of 
the estimator around the detectability threshold alluded to earlier. Specifically, we observe that if the eigen-SNR 
of a signal is above a critical value then reliable detection using the new algorithm is possible. Conversely, if the 
eigen-SNR is below the critical value then the algorithm, correctly for the reason described earlier, is unable to 
distinguish the signal from noise. 

The paper is organized as follows. We formulate the problem in Section |II] and state the main result in Section 
Hm The effective number of signals is defined in Section IIII-AI along with a discussion on its implications for 
applications such as array processing, sensor networks and machine learning. A new algorithm for detecting the 
number of signals is presented in Section |IVl Concluding remarks are offered in Section |Vl The mathematical 
proofs of the main result are provided in Section |Vll 

II. Problem formulation 

We observe m samples ("snapshots") of possibly signal bearing n-dimensional snapshot vectors xi , . . . , 
where for each i, the snapshot vector has a (real or complex) multivariate normal distribution, i.e., Xj ~ A/'n(0, R) 
and the Xj's are mutually independent. The snapshot vectors are modelled as 



where Zj ~ A/'n(0, S), denotes an n-dimensional (real or complex) Gaussian noise vector where the noise covariance 
S may be known or unknown, Sj ~ A/'a;(0, R^) denotes a fc-dimensional (real or complex) Gaussian signal vector 
with covariance R^, and A is a n x unknown non-random matrix. Since the signal and noise vectors are 
independent of each other, the covariance matrix of Xj can hence be decomposed as 



Xj — A Sj -|- Zj 



for i = 1, . . . , m 



(1) 



R = * + S 



(2) 



where 



* = AR^A', 



(3) 



with ' denoting the complex conjugate or real transpose. Assuming that the matrix A is of full column rank, i.e., the 
columns of A are linearly independent, and that the covariance matrix of the signals R<j is nonsingular, it follows 
that the rank of ^ is k. Equivalently, the n — k smallest eigenvalues of ^ are equal to zero. 
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If the noise covariance matrix XI were known apriori and was non-singular, a "noise whitening" transformation 
may be applied to the snapshot vector Xj to obtain the vector 

= S"i/2xi, (4) 

which will also be normally distributed with covariance 

Rs := = S"^* + I. (5) 

Denote the eigenvalues of Rs by Ai > A2 > • • • > A„. Recalling the formulation of the generalized eigenvalue 
problem [l][Section 8.7], we note that the eigenvalues of Rs are exactly the generalized eigenvalues of the regular 
matrix pair (R, H). Then, assuming that the rank of S~^* is also k, it follows that the smallest n — k eigenvalues 
of Rs or, equivalently, the generalized eigenvalues of the matrix pair (R, S)), are all equal to 1 so that 

Afc+i = Afc+2 = ■ • • = A„ = A = 1, (6) 

while the remaining k eigenvalues Rs of will be strictly greater than one. 

Thus, if the true signal-plus-noise covariance matrix R and the noise-only covariance matrix S were known 
apriori, the number of signals k could be trivially determined from the multiplicity of the eigenvalues of Rs 
equalling one. 

The problem in practice is that the signal-plus-noise and the noise covariance matrices R are unknown so that 
such a straight-forward algorithm cannot be used. Instead we have an estimate the signal-plus-covariance matrix 
obtained as 

^ m 

R=-Vx,x^ (7) 
m ^-^ 

1=1 

and an estimate of the noise-only sample covariance matrix obtained as 

N 



1 

N ^ 



zjz'j (8) 



where Xj for i = 1, . . . , m are (possibly) signal-bearing snapshots and Zj for j = 1, . . . , are independent noise- 
only snapshots. We assume here that the number of noise-only snapshots exceeds the dimensionality of the system, 
i.e., N > n + 1, so that the noise-only sample covariance matrix S, which has the Wishart distribution [11], 
is non-singular and hence invertible with probability 1 [12, Chapter 3, pp. 97], [13, Chapter 7.7, pp. 272-276]. 
Following dSjl, we then form the matrix 

Rg = £-iR, (9) 

and compute its eigen-decomposition to obtain the eigenvalues of Rg, which we denote by Ai > A2 > • • • > A„. 
We note, once again, that the eigenvalues of Rg are simply the generalized eigenvalues of the regular matrix pair 
(R, S). Note that whenever N < n, the signal-plus-noise sample covariance matrix R will be singular so that the 
n — N generalized eigenvalues will equal zero, i.e., Xn+i = = • • • = A„ = 0. Figure [T] illustrates why the 

blurring of the sample eigenvalues relative to the population eigenvalues makes the problem more challenging. 

In this paper, we are interested in the class of algorithms that infer the number of signals buried in arbitrary 
noise from the eigenvalues of Rg or Rs alone. Such algorithms are widely used in practice and arise naturally 
from classical multivariate statistical theory [10] where the matrix Rg is referred to as the multivariate F matrix 
[12], [14]. The information theoretical approach to model order estimation, first introduced by Wax and Kailath 
[6], was extended to the colored noise setting by Zhao et al in [15] who prove consistency of their estimator in the 
large sample size regime; their analysis does not yield any insight into the finite sample setting. 

Consequently, research has focussed on developing sophisticated techniques for improving performance of 
eigenvalue based methods in the finite sample setting. Zhu et al [16] improve the performance of their eigenvalue 
estimator by assuming a model for the noise covariance matrix. Stoica and Cedervall [17] improve the performance 
of their estimator in two reasonable settings: one, where it is reasonable to assume that the noise covariance matrix 
is block diagonal or banded and two, where the temporal correlation of the noise has a shorter length than the 
signals. Other techniques in the literature exploit other characteristics of the signal or noise to effectively reduce the 
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Fig. 1: The dimension of the "noise" subspace is equal to the multiplicity of the population eigenvalue equal to 
one. When the population eigenvalues are known, then detecting the number of signals becomes trivial. However, 
estimating the number of signals from the sample generalized eigen-spectrum is considerably more challenging 
because of the finite sample effects. Specifically, the finite number of noise-only and signal-plus-noise samples 
induces a blurring in the sample eigenspectrum relative to the population eigenspectrum makes discrimination of 
the "signal" from the "noise" challenging. The figure shows one random instance generated for a n = 20 dimensional 
system with = 25 noise-only samples and m = 40 signal-plus-noise bearing samples. 



dimensionality of the signal subspace and improve model order estimation given finite samples. See for example 
[18], [19] and the references in [9]. 

Informally speaking, it is evident that performance of such model order estimation algorithms is coupled to the 
"quality" of the estimated signal-plus-noise and noise-only covariance matrices which in turn are dependent on 
the number of snapshots used to estimate them, respectively. Researchers applying these techniques have noted 
the absence of a mathematically rigorous, general purpose formula in the Uterature for predicting the minimum 
number of samples needed to obtain "good enough" detection accuracy (see, for example [3] [pp. 846]. A larger, 
more fundamental question that has remained unanswered, till now, is whether there is a statistical limit being 
encountered. 

We tackle this problem head on in this paper by employing sophisticated techniques from random matrix theory 
in [20]. We show that in an asymptotic sense, to be made precise later, that only the "signal" eigenvalues of Rs 
that are above a deterministic threshold can be reliably distinguished from the "noise" eigenvalues. The threshold is 
a simple, deterministic function of the the dimensionality of the system, the number of noise-only and signal-plus- 
noise snapshots, and the noise and signal-plus noise covariance, and described explicitly next. Note the applicability 
of the results to the situation when the signal-plus-noise covariance matrix is singular. 

III. Main result 

For a Hermitian matrix A with n real eigenvalues (counted with multiplicity), the empirical distribution function 
(e.d.f.) is defined as 

„A , , Number of eigenvalues of A < x 

F^{x) = —. (10) 

n 

Of particular interest is the convergence of the e.d.f. of Rg in the signal-free case, which is described next. 



SAMPLE GENERALIZED EIGENVALUE BASED DETECTION 



6 



Theorem 3.1: Let Rg denote the matrix in Q formed from m (complex Gaussian) noise-only snapshots and N 
independent noise-only (complex Gaussian) snapshots. Then the e.d.f. F^^{x) F^^{x) almost surely for every 

X, as m, n{m) oo, m, N{m) — > oo and Cm = n/m ^ c > and c]y = n/N — > ci < 1 where 



where 



\bi,bo]{x) = 1 when 6i < X < 62 and zero otherwise, and 5{x) is the Dirac delta function. 

Proof: This result was proved in [14]. When ci ^ we recover the famous Marcenko-Pastur density [21]. 

■ 

The following result exposes when the "signal" eigenvalues are asymptotically distinguishable from the "noise" 
eigenvalues. 

Theorem 3.2: Let Rg denote the matrix in ^ formed from m (real or complex Gaussian) signal-plus-noise 
snapshots and independent (real or complex Gaussian) noise-only snapshots. Denote the eigenvalues of Rs by 
Ai > A2 > • • • > Afc > \k+i = . . . A„ = 1. Let Ij denote the j-th largest eigenvalue of Rg. Then as n, m(n) 00, 
n, N{n) — > 00 and Cm = n/m — > c > and cjy = n/N — > ci < 1 we have 



ci Xj - Aj + 1 + Jci^Xj - 2 ci A/ - 2 ci Xj + A/ - 2 Xj + I 

Xj \ l-c- c ^ ^—j- I , Xj > r(c, ci) 

cic + C+1+ ci + 2 i/c + ci - cic , ^ / X 

, Aj < t(c,ciJ 



ci^-hl-2ci 

foT j = I, . . . , k and the convergence is almost surely and the threshold T(c, ci) is given by 



T(c,ci) = , (13) 

where 

^ _ (1 + ci)a + V^V4a - ci + (1 - cQ^c^ _ (1 + ci)a + Va(2ci + c(l - ci)) ^^^^ 
(1 — ciYa (1 — ci)2a 

and a = c + ci — cic. 

Proof: The result follows from Theorem 16.51 The threshold T(c, ci) is obtained by solving the inequality 

t'>T 

where for j = 1, . . . , /c, t', from [9], [22]-[24], is given by 

t' = - ' 



A,(l+ 



A,-l 



and T is given by ([30] ). 

Note that when ci ^ 0, T(c, ci) ^ (1 + t/c) so that we recover the results of Baik and Silverstein [23]. 
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Fig. 2: Plot of the minimum (generalized) Eigen-SNR required (equal to T(c, ci) — 1 where T(c, ci) is given by 
([T3] )) to be able to asymptotically discriminate between the "signal" and "noise" eigenvalue of the matrix Rg 
constructed as in (|9ll as a function of the ratio of the number of sensors to snapshots for different values of 1/ci 
where ci Number of sensors/Number of noise-only snapshots. The gap between the upper two lines and the 
bottom most line represents the SNR loss due to noise covariance matrix estimation. 

A. Effective number of identifiable signals 

Theorem 13 .21 brings into sharp focus the reason why, in the large-system-relatively-large-sample-size limit, model 
order underestimation is sometimes unavoidable. This motivates our heuristic definition of the effective number of 
identifiable signals below: 

fce//(R, 5]) = # Eigs. of S-iR > T(c, ci) « T (-, ^) . (15) 

If we denote the eigenvalues of Rs = S~^R by Ai > A2 > • • • > > A^+i = . . . A„ = 1 then we define 
the eigen-SNR of the j-th signal as Xj — 1 then (fTSl) essentially states that signals with eigen-SNR's smaller than 
T{n/m,n/N) will be asymptotically undetectable. 

Figure |2] shows the eigen-SNR threshold T(c, ci) — 1 needed for reliable detection for different values as a function 
of c for different values of 1/ci . Such an analytical prediction was not possible before the results presented in this 
paper. Note the fundamental limit of detection in the situation when the noise-only covariance matrix is known 
apriori (solid line) and increase in the threshold eigen-SNR needed as the number of snapshots available to estimate 
the noise-only covariance matrix decreases. 

B. Implications for array processing 

Suppose there are two uncorrelated (hence, independent) signals so that R^ = diag((Tg^, cjgg). In ([T|) let A = 
[V1V2]. In a sensor array processing application, we think of vi = v(6'i) and V2 = V2(6'2) as encoding the array 
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manifold vectors for a source and an interferer with powers a^^ and dgj, located at 9i and 62, respectively. The 
signal-plus-noise covariance matrix is given by 

R = fJsiViv'i + 0-32 V2V2 + 5] (16) 

where I] is the noise-only covariance matrix. The matrix Rs defined in ([5]) can be decomposed as 

Rs = T.^R = criiS^Viv'i + S;-Vi2V2V^ + I 

so we that we can readily note that Rs has the n — 2 smallest eigenvalues A3 = . . . = A„ = 1 and the two largest 
eigenvalues 



(f^Sl +^S2 I|u2f) , V(^S1 -^S2 l|u2P)' + 4<j2^a|2Kui,U2)|2 

Ai - i H \ (1/a) 



(aiJIuif +^12 ||U2||2) ^(aiJIuiP -ai2 ||u2P)' + 4<j2^(ji2Kui,U2)|2 

A2 = 1 H (1/b) 

respectively, where ui := 'S^^^'^vi and U2 := 'S~^^'^V2 . Applying the result in Theorem 13.21 allows us to express 
the effective number of signals as 

'2 ifT(^,^)<A2 
'€efr = <! 1 if A2<T(^,^) <Ai (18) 

ifAi<T(^,f). 

Equation (fTSl ) captures the tradeoff between the identifiability of two closely spaced signals, the dimensionality 
of the system, the number of available snapshots and the cosine of the angle between the vectors vi and V2. Note 
that since the effective number of signals depends on the structure of the theoretical signal and noise covariance 
matrices (via the eigenvalues of Re), different assumed noise covariance structures (AR(1) versus white noise, for 
example) will impact the signal level SNR needed for reliable detection in different ways. 

C. Other applications 

There is interest in detecting abrupt change in a system based on stochastic observations of the system using 
a network of sensors. When the observations made at various sensors can be modeled as Gauss-Markov random 
field (GMRF), as in [25], [26], then the conditional independence property of GMRF's [27] is a useful assumption. 
The assumption states that conditioned on a particular hypothesis, the observations at sensors are independent. This 
assumption results in the precision matrix, i.e., the inverse of the covariance matrix, having a sparse structure with 
many entries identically equal to zero. 

Our results might be used to provide insight into the types of systemic changes, reflected in the structure of the 
signal-plus-noise covariance matrix, that are undetectable using sample generalized eigenvalue based estimators. 
Specifically, the fact that the inverse of the noise-only covariance matrix will have a sparse structure means that 
one can experiment with different (assumed) conditional independence structures and determine how "abrupt" the 
system change would have to be in order to be reliably detected using finite samples. 

Spectral methods are popular in machine learning applications such as unsupervised learning, image segmentation, 
and information retrieval [28]. Generalized eigenvalue based techniques for clustering have been investigated in 
[29], [30]. Our results might provide insight when spectral clustering algorithms are likely to fail. In particular, we 
note that the results of Theorem 13.21 hold even in situations where the data is not Gaussian (see Theorem 16.5b as 
is commonly assumed in machine learning applications. 
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IV. An algorithm for reliable detection of signals in noise 

In [10], Johnstone proves that in the signal-free case, the distribution of the largest eigenvalue of Rg, on 
appropriate centering and scaling, can be approximated to order 0(n~^/'^) by the Tracy-Widom law [31]-[33]. In 
the setting where there are signals present, we expect, after appropriate centering and scaling, the distribution of 
the signal eigenvalues of Rg above the detectability threshold will obey a Gaussian law whereas those below the 
detectability threshold will obey the Tracy-Widom law as in the signal-free case. An analogous results for the signal 
bearing eigenvalues of Rs was proved by Baik et al [22] and El Karoui [34]. Numerical investigations for (see 
Figure (3]) corroborate the accuracy of our asymptotic predictions and form the basis of Algorithm 1 presented below 
for estimating the number of signals at (asymptotic) significance level a. Theoretical support for this observation 
remains incomplete. 

Algorithm 1 

Input: Eigenvalues \j for j = 1, . . . , n of 

1. Initialization: Set significance level a € (0, 1) 

2. Compute Tc := T'W^^^-^{\ - a) from Table HH 

3. Set k = 

4. Compute ^{r,c}['^ ~ k,m] and cr{M,c}['^ ~ k,m] from Table [(a)] 

\ogmXk+i/N - inTst^c}[n - k,m - k,N] 

<^{m.c}['n - k,m ~ k,N\ 

6. If yes, then go to step 9 

7. Otherwise, increment k. 

8. If fc < min(n, ?n), go to step 3. Else go to step 9. 

9. Return fc = fc 



Figure |4] illustrates the accuracy of the predicted statistical limit and the ability of the proposed algorithm to 
reliably detect the presence of the signal at this limit. 

In the special setting where the noise covariance matrix is known apriori, the results of Baik et al [22], El 
Karoui [34] and Ma [35] form the basis of Algorithm 2 presented below for estimating the number of signals at 
(asymptotic) significance level a. 
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TABLE I: Parameters for signal detection algorithms. 
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Q 


1 - a 


rM/,7^(l-a) 


rwc^(i-a) 


0.990000 


0.010000 


-3.89543267306429 


-3.72444594640057 


0.950000 


0.050000 


-3.18037997693774 


-3.19416673215810 


0.900000 


0.100000 


-2.78242790569530 


-2.90135093847591 


0.700000 


0.300000 


-1.91037974619926 


-2.26618203984916 


0.500000 


0.500000 


-1.26857461658107 


-1.80491240893658 


0.300000 


0.700000 


-0.59228719101613 


-1.32485955606020 


0.100000 


0.900000 


0.45014328905825 


-0.59685129711735 


0.050000 


0.950000 


0.97931605346955 


-0.23247446976400 


0.010000 


0.990000 


2.02344928138015 


0.47763604739084 


0.001000 


0.999000 


3.27219605900193 


1.31441948008634 


0.000100 


0.999900 


4.35942034391365 


2.03469175457082 


0.000010 


0.999990 


5.34429594047426 


2.68220732168978 


0.000001 


0.999999 


6.25635442969338 


3.27858828203370 



TABLE II: The third and fourth column show the percentiles of the Tracy-Widom real and complex distribution 
respectively corresponding to fractions in the second column. The percentiles were computed in MAT LAB using 
software provided by FoUcmar Bornemann for the efficient evaluation of the real and complex Tracy-Widom 
distribution functions Fj^'^|(x). The percentiles are computed using the fzero command in MAT LAB . The 
accuracy of the computed percentiles is about ±5 x 10^^^ in absolute error terms. 



Algorithm 2 

Input: Eigenvalues \j for j = 1, . . . , n of Rs 

1. Initialization: Set significance level q £ (0, 1) 

2. Compute Tq := TW'^^f.-^{l — a) from Table llll 

3. Set k = 

4. Compute A'{r,c}['^ ~ fc,m] and cr^g cii^i- — fc,m] from Table [(b)] 

- ^ mAfc+i - M{K,C} [n - k, m] 

J. is ■ - - Ta . 

cr{R,c}[n - k,m\ 

6. If yes, then go to step 9 

7. Otherwise, increment k. 

8. If k < min(n, m), go to step 3. Else go to step 9. 

9. Return k = k 
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X (= Test Statistic) 



(a) Here cr^ = 0.5, so that Ai = 1 + = 1.5 < T(320160, 320960) = 3.4365. 



Empirical CDF 




0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 

X (= Test Statistic) 

(b) Here = 5, so that Ai = 1 + = 6 > T(f|2, |§) = 3.4365 

Fig. 3: In (a), for the setting described in Theorem 13.21 we set n = 320, m = 160, N = 960, ci^ = 0.5, and w.l.o.g. 
5] = I, R = diag(Ai = 1 + o"^, 1, . . . , 1) and compare the the empirical cdf of the largest eigenvalue of Rg with 
the largest eigenvalue of Rg with R = I, i.e., in the noise-only case, over 1000 Monte-Carlo trials. In (b), we plot 
the empirical cdf but now with cr^ = 5. 
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#Sensors = 320, #Noise-only Snapshots = 960, Prob(False Detection | No Signal) = 0.01 




0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 

#Sensors/#Signal-plus-Noise Snapshots 



Fig. 4: A heat map of the log probabiUty of signal detection using Algorithm 1 in Section |Tvl with the significance 
level a set at 0.01, in (eigen) SNR versus number of sensors to number of signal-plus-noise snapshots phase 
space. In this example, for the setting described in Theorem 13.21 we set n = 320, = 960 and w.l.o.g. S = I, 
R = diag(Ai = 1 + SNR, 1, . . . , 1) and evaluated Fmh{k = 1) over 1000 Monte-Carlo trials and a grid of 100 
equally spaced points in the -5 dB to 15 dB (eigen) SNR range and 100 equally spaced points in the ci = n/m 
space by setting m = n/ci. The values of the colormap at each of the 100 x 1000 faces were interpolated across 
each line segment and face to obtain the above plot. In the dark zone (upper half of the plot) a signal can be 
reliably detected whereas in the lighter zone (lower half of the plot) the signal is statistically indistinguishable from 
noise as evidenced from the probability of detection being close to the significance level. The superimposed solid 
black line demarcates the theoretically predicted threshold while the superimposed solid red line is the theoretically 
predicted threshold in the setting where the noise covariance matrix is perfectly known. The gap between the two 
lines thus represents the SNR loss due to noise covariance matrix estimation. 
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V. Conclusion 



Figure |4] captures the fundamental statistical limit encountered when attempting to discriminate signal from noise 
using finite samples. Simply put, a signal whose eigen-SNR is below the detectability threshold cannot be reliably 
detected while a signal above the threshold can be. In settings such as wireless communications and biomedical 
signal processing where the signal power is controllable, our results provide a prescription for how strong it needs 
to be so that it can be detected. If the signal level is barely above the threshold, simply adding more sensors might 
actually degrade the performance because of the increased dimensionality of the system. If, however, either due 
to clever signal design or physics based modeling, we are able to reduce (or identify) the dimensionality of the 
subspace spanned by signal, then according to Figured the detectability threshold will also be lowered. With VLSI 
advances making sensors easier and cheaper to deploy, our results demonstrate exactly why the resulting gains 
in systemic performance will more than offset the effort we will have to invest in developing increasingly more 
sophisticated dimensionality reduction techniques. Understanding the fundamental statistical limits of techniques 
for signal detection in the setting where the noise-only sample covariance matrix is singular remains an important 
open problem. 
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A. Mathematical preliminaries 

Let for i,j = 1,2,..., Xij be a collection of complex valued i.i.d. random variables with EXu = and 
E|Xi 1 p = 1. For positive integers n and m let X„ = (Xij), i = 1,2, . . . ,n, j = 1,2, . . . ,m. Assume for each n 
Tn is an n X n Hermitian nonnegative definite matrix. The matrix 



where T„ is any Hermitian square root of T„, can be viewed as a sample covariance matrix, formed from m 

1 /2 

samples of the random vector T„ X.i with X.i denoting the first column of X„, which has T.„ for its population 
covariance matrix. When n and m are both large and on the same order of magnitude, B„ will not be near T„, 
due to an insufficient number of samples required for such a large dimensional random vector. However, there exist 
results on the eigenvalues of B„. They are limit theorems as n ^ oo with m = m{n) and c„ = n/m — > c, which 
provide information on the eigenvalues of T„. One result [36] is on the empirical distribution function (e.d.f.), F^", 
of the eigenvalues of B„, which throughout the paper, is defined for any Hermitian n x n matrix A as 



The limit theorem is expressed in terms of the Stieltjes transform of the limiting e.d.f. of the F " 's, where for 
any distribution function (d.f.) G its Stieltjes transform, inc, is defined to be 
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VI. Appendix 



B„ = (1/m) 



rpl/2"V" V^'T'l/^ 



F^{x) = (l/n)(number of eigenvalues of A < x). 
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There exists a one-to-one correspondence between the distribution functions (d.f.'s) and their Stieltjes transforms, 
due to the inversion formula 

b 



G{h) — G{a) = lim / '^mcix + iv)dx 



for a,b continuity points of G. 

The limit theorem allows the T„ to be random, only assuming as n — > cxd, the convergence of F^" to a nonrandom 
proper probability distribution function Hn, i.e., Hn = F^" H. The theorem states that with probability one, 
as n ^ oo, F^" — F, where F is nonrandom, with Stieltjes transform m = mpiz), z £ C+ satisfying the 
equation 

m = / — i ^ dH(t), (19) 

J r(l — c — czm) — z 

which is unique in the set {m G C : — + cm G C"*"}. 

It is more convenient to work with the eigenvalues of the m x m matrix (l/m)X„T„Xn, whose eigenvalues 
differ from those of B„ by \n — m\ zero eigenvalues. Indeed, with Ia denoting the indicator function on the set A 
we have the exact relationship 

^(i/™,)x;t„x„(^) = (1 - c„))/[o,oo)(x) + c„F^"(x) 
-^D= (1 - c))/[o,oo)(x) + cF{x) ^ F^'^(x) 

almost surely, implying 

mpc.ni^z) = —{1 — c)/z + cmp^z). (20) 
Upon substituting mpa,H into ([T9l l we find that for z G C+ m = mpa,H{z) solves the equation 

z = -- + c I -^—dH{\), (21) 
m J 1 + Am 

and is unique in C^. Thus we have an explicit inverse for nipc.H. 

Qualitative properties of F'^'^ have been obtained in [37], most notably the fact that on (0, cxd) F'^'^ has a 
continuous derivative. The paper [37] also shows how intervals outside the support of F'^'^ can be determined from 
the graph of (|2TI) for m G M. 

Let Sg denote the support of the d.f. G, S'q its complement, and define Xc^h = Xc^nirn) to be (|2TI ) with m G M. 
Intuitively, on Sp^,H nipcH is well defined and increasing. Therefore it is invertible on each interval in S'pa,H, its 
inverse, namely Xc^h, is also increasing. The details are stated in the following. 

Lemma 6.1 (Theorems 4.1, 4.2 of [37]): If x G Sp^,„, then m = mpa,H satisfies (1) m G M\{0}, (2) -m~^ G 
S'j^, and (3) -£^Xc,H{m) > 0. Conversely, if m satisfies (l)-(3), then x = Xcnim) G S'p^_H. 

In simple terms S'p^,„ is comprised of the range of values where Xc,h is increasing. 

Another result which will be needed later is the following. 

Lemma 6.2 (Theorem 4.3 of [37]): Suppose each m contained in the interval [mi,m2] satisfies (1) and (2) of 
Lemma [STT] and -^Xc,H{n^i) > for i = 1,2. Then ^Xc,_f/("i) > for all m G {mi, 1712). 
Limiting eigenvalue mass at zero is also derived in [37]. It is shown that 

F(0) = | ^(1-^(0))^!' (22) 

^ ^ \l-c-\ c(l-F(0)) > L 

B. Support of eigenvalues 

Since the convergence in distribution of F^" only addresses how proportions of eigenvalues behave, understanding 
the possible appearance or non-appearance of eigenvalues in S'p^__H requires further work. 

The question of the behavior of the largest and smallest eigenvalues when T„ = I has been answered by Yin, 
Bai, and Krishnaiah in [38], and Bai and Yin in [39], respectively, under the additional assumption E|Xii|^ < co: 
the largest eigenvalue and min(n, m)* largest eigenvalue of (l/m)X„X* converge a.s. to (1 + -v/c)^ and (1 — -v/c)^ 
respectively, matching the support, [(1 — ^/c)^, (1 + \/c)^] of F on (0, cxd). More on F when T„ = I will be given 
later. 
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For general T„, restricted to being bounded in spectral norm, the non-appearance of eigenvalues in 5^^,^ has 
been proven by Bai and Silverstein in [40]. Moreover, the separation of eigenvalues across intervals in S'p^_H, 
mirrors exactly the separation of eigenvalues over corresponding intervals in S'^ [41]. The results are summarized 
below. 

Theorem 6.1: Assume additionally E|Xii|^ < oo and the T„ are nonrandom and are bounded in spectral norm 
for all n. 

Let F'^"'^" denote the "limiting" e.d.f. associated with (l/m)X*T„X„, in other words, F'^"'^" is the d.f. having 
Stieltjes transform with inverse (|2T]) . where c,H are replace by Cn,Hn- 
Assume the following condition: 

• (*) Interval [a, h] with a > lies in an open interval outside the support of F'^"'^" for all large n. 
Then P(no eigenvalue of B„ appears in [a, b] for all large n) = 1. 

For nxn Hermitian non-negative definite matrix A, let denote the /c* largest eigenvalue of A. For notational 
convenience, define A^^ = oo and X^+i = 0. 

(i) If c(l — H{0)) > 1, then xq, the smallest value in the support of F'^'^ , is positive, and with probability 1, 
A^" — > x'' as n — > CO. 

(ii) If c(l - H{0)) < 1, or c(l - H{0)) > 1 but [a, b] is not contained in [0, xq], then mF-=:H{b) < 0, and for all 
n large there is an index i„ for which 

Xj^ > -l/mFc.H{b) and Xj^_^^ < -l/mpcHia). (23) 

Then P{Xf^" > b and A^^"^^ < a for all large n) = 1. 

Proof: See proof of Theorems 1.1 in [40], [41]). ■ 

The behavior of the extreme eigenvalues of (l/m)X„X* leads to the following corollary of Theorem 16. II 

Corollary 6.2: If A^" converges to the largest number in the support of H, then Af " converges a.s to the largest 
number in the support of F. If A^" converges to the smallest number in the support of H, then c < 1 (c > 1) 
implies A^" (^(^^Z™)"'^"^" converges a.s. to the smallest number in the support of F {F^'^). 

In Theorem 16.11 Case (i) applies when n > m, whereby the rank of B„ would be at most m, the conclusion 
asserting, that with probability 1, for all n large, the rank is equal to m. From Lemma [67T1 Case (ii) of Theorem 
16. II covers all intervals in Sp^,H on (0, oo) resulting from intervals on (— oo, 0) where Xc,h is increasing. For all n 
large Xc„,h„ is increasing on [mi?c„,ff„ (a), mi?c„,ff„ (6)], which, from inspecting the vertical asymptotes of Xc„,_f/„ 
and Lemma 16711 must be due to the existence of Xj", Xj"^^ satisfying (1231 ). 

Theorem 16. II easily extends to random T.„, independent of {Xjj : i,j > 1} with the aid of Tonelli's Theorem 
[42, pp. 234], provided the condition (*) on [a,b] is strengthened to: 

• (**) With probability 1 for all n large [a, b] (nonrandom) lies in an open interval outside the support of F*^'"^". 
Indeed, let T denote the probability space generating {T„}, X the probability space generating {Xij : i,j > 1}. Let 
their respective measures be denoted by Pt,Px, the product measure on T x X by PtxX- Consider, for example 
in case (ii), we define 

^ = {^Z" > ^ '^f +1 < ^ 
Let t E r be an element of the event defined in (**). Then by Theorem 16. II I a ((t, x)) = 1 for all x contained in 
a subset of X having probability 1. Therefore, by Tonelli's theorem 

P{A) = j lAit,x)dPT^xi{t,x)) = j j lAit,x)dPx{x) dPrit) = J IdPrit) = 1. 

Consider now case (ii) of Theorem 16.11 in terms of the corresponding interval outside the support of H and the 
HnS. By Lemma |6?T] and condition (*), we have the existence of an e > such that ^ [mi^o w (a) — e, mi?c,ff (6)+e], 
and for all n large 

-^Xc„,_f/„("i) = ^ (l-Cn I ,}^^^^ dHniXyl > 0, me [mFo.ff(a) - e,mF'^.H{b) + e]. (24) 
dm \ J [1 + Am)^ J 

Let ta = — l/m^c,ff (a), 4 = — 1/m^c.ff (6). Then by Lemma |6?T] we have the existence of an e' > for which 
ta — e' > and [ta — e', tb + e'] C S'jj for all n large. Moreover, by (l24l ) we have for all n large 

A^ 

-.dHniX)<l, t£[ta-€',tb + e']. (25) 

[x-ty 
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Necessarily, A^" > tf, + e' and Xj'"^i < ta — e' ■ 

Notice the steps can be completely reversed, that is, beginning with an interval [ta-,th\, with ta > 0, lying in an 
open interval in 5^ for all n large and satisfying (|25] ) for some e' > 0, will yield [a,b], with a = Xc,H{—^/ia), 
b = Xc^H{—^/tb), satisfying condition (*). Case (ii) applies, since [a, b] is within the range of Xc,_f/(m) for m < 0. 
If c(l — H{0)) > 1, then we would have a > xq. 



C. Behavior of spiked eigenvalues 

Suppose now the T„'s are altered, where a finite number of eigenvalues are interspersed between the previously 
adjacent eigenvalues \J'\i and Xj". It is clear that the limiting F will remain unchanged. However, the graph 
of Xc„,H„ on (— 1/Af"_|_^, — l/A^") will now contain vertical asymptotes. If the graph remains increasing on two 
intervals for all n large, each one between successive asymptotes, then because of Theorem 16.11 with probability 
one, eigenvalues of the new B„ will appear in S'p^^H for all n large. 

Theorem 16.31 below shows this will happen when a "sprinkled", or "spiked" eigenvalue lies in (tai^b)- Theorem 
16.41 provides a converse, in the sense that any isolated eigenvalue of Bn must be due to a spiked eigenvalue, the 
absence of which corresponds to case (ii) of Theorem 16.11 

Theorem 16.31 below, allows the number of spiked eigenvalues to grow with n, provided it remains o(?i). 

Theorem 6.3: Assume in additon to the assumptions in Theorem 16. II on the and T„: 

(a) There are £ = o{n) positive eigenvalues of T„ all converging uniformly to t' , a positive number. Denote by 
Hn the e..d.f. of the n — I other eigenvalues of T„. 

(b) There exists positive ta < 4 contained in an interval (a, /3) with a > which is outside the support of Hn 
for all large n, such that for these n 

fort = ta,tb. 

(C) t' G {ta,tb). 

rp rp 

Suppose A^ Aj "j^^^i are the eigenvalues stated in (a). Then, with probability one 

lim Af " = • • • = lim Af " = t' ( I + c f -^dH{X)] . (26) 



n'^'So' ir.+i-l - ' ^ J t' - X 

Proof: For m € [-1/ia, -l/ife] n {-1/t'Y, we have 

ir,+e-i ^T, 

Xc^^hA^) = — - + c„, I - ^-j^ h - — - I " dHn{X) 



1 / 1 '"^^^ X^ n-e f X , ~ , 
m \n ^ H-A"m n J l + Xm 



By considering continuity points of H in (a, (3) we see that H is constant on this interval, and consequently, 
this interval is also contained in S'^. 

Because of (b) we have -^Xc^nim) > for m = -1/ia, -l/ifo (recall (l24l).(|25l)). 

By Lemma [6^ we therefore have ^Xc,//(m) > for all m G (— 1/ta, —l/tb)- Thus we can find [t^, t^] C [ta-, 4] 
and (5 > 0, such that t' G {ta-itb) for all n large fj (m) > 6 for all m G [— — l/t^]. 

It follows that for any positive e sufficiently small, there exist positive 5' with 5' < e, such that, for all n large, 
both l-l/t' - e - y, -l/t' - e], and [-1/t' + e, -1/t' + e + 5']: 

• 1) are contained in [—l/ia, — /tf^], and 

• 2) ^Xc^^//^(m) > for all m contained in these two intervals. 

Therefore, by Lemma 16.11 for all n large, [xc„ — e — 5'), Xc„,_f/„ — e)] and [xc„,/i-„(— + 
e),Xc^^H„{-^/t' + e + 6')] lie outside the support of F""^^". Let ol = Xc,H(-l/t' - e - §5'), 6l = Xc,H{-l/t' - 
e - i(5'), an = Xc,H{-Vi' + e + \^')^ and bn = Xc,H{-^/t' + e + §5'). Then for all n large 

[aL,6L] C {x,,H{-l/t' -e-l6'),x,,H{-l/t' -e-\5')) 

C K,j^„(-l/t'-e-<5'),x,„,H„(-l/t'-e)] (27) 
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and 

[aR,bR] C {xc,Hi-l/t' + e + ^6'),Xc,H{-l/t' + e + ^6')) 

C [xe„,H„(-lA' + e),a;c„,H„(-lA' + e + 5')]- (28) 

It follows then that [a^, 6/,], [a^, bf(] each lie in an open interval in S'p^,^^^ for all n large. Moreover mire, ff (ft^j) < 
0. Therefore, case (ii) of Theorem 16. II applies and we have 

P{Xf" < an and A^"^^_^ > bi for all n large) = 1. 

Therefore, considering a countable collection of e's converging to zero, we conclude that, with probability 1 

lim Af" = lim Af" = x,,H(-l/t') = 

n— >oo " n^oo 'n^^ 

■ 

Theorem 6.4: Assume, besides the assumptions in Theorem 16. II there is an eigenvalue of B„ which converges in 
probability to a nonrandom positive number. A' G S'p. Let interval [a, b] G S'p, with a > 0, be such that A' G (a, b), 
and let ta = —l/mc,H{o), i' = — l/^n-c./zlA'), tt, = —l/mc,H{b) (finite by Lemma [6TT] ). Then < ta < t' < tb, 
implying (c) of Theorem 16.31 Let £ = i{n) denote the number of eigenvalues of T„ contained in [ta, tb] and let Hn 
denote the e.d.f. of the other n — £ eigenvalues of T„. Then £ = o{n) and (b) of Theorem 16. 3 1 is true. If £ remains 
bounded, then (a) of Theorem 16.31 also holds. 

Proof: By Lemma ISTT] [ta, tb] G 5^, and for a suitable positive e, x^^h is increasing on [mc^j{{a) — e, mc^H{b) + 
e], which does not contain 0. 

Therefore ta < t' < tb- If c(l — H{0)) > 1, that is, case (i) of Theorem 16.11 holds, then a > xq, since xq is the 
almost sure limit of A^" so A' cannot be smaller than it, and necessarily xq G Sp. Therefore mc^nib) < 0, so that 

< ta- 

It must be the case that only o(n) eigenvalues of tn lie in [ta;4]> since otherwise [ta^ifo] would not be outside 
the support of H. We have then Hn — .ff as n — > oo, so from the dominated convergence theorem we have 
^^c„,H„('") ^ for all m G [mc,H{a) - e,mc,H{b) + e], implying for all n large -£^x^^ fj^{m) > 

for all m G [mc^H{o),mc^H{b)]. Therefore (b) is true 

We assume now that £ is bounded. Suppose (a) does not hold. Then we could find a subsequence {uj} of the 
natural numbers for which £' = £'{n) of the £ eigenvalues converge to a t' 7^ t' , the remaining £ — £' , if positive, 
eigenvalues remaining a positive distance d from t!. Replace {T„} with {T^} which matches the original sequence 
when n = rij and for n 7^ rij, has £' eigenvalues equal to tf, with the remaining £ — £' , again, if positive, 
eigenvalues of at least d away from t'. Then we have by Theorem 16.11 (l26l ). with t' replaced by t', holding 
for £' of the eigenvalues of (l/m)T;^/^X„X;r;;^/l Thus, on {uj}, we have the almost sure convergence of £ 
eigenvalues of B„ to Xc,//(— G [a,b] which, because Xc,_f/(— is an increasing function, does not equal 
A' = Xc,h(— This contradicts the assumption of convergence in probability to eigenvalues to only one number, 
namely A'. Therefore (a) holds. ■ 

D. Behavior of extreme eigenvalues 

Consider now t' lying on either side of the support of H. Let A™"^ and A™^^ denote, respectively, the smallest 
and largest numbers in the support of Hn Notice that gn{i) = c„ / jj^-j^dHn{t) is decreasing for t > X^^^, and 

if A™° > 0, gn is increasing on iO, XT''). 

Therefore, if for all n large, t' > X^^^, it is necessary and sufficient to find a t^ G (A™'^^, t') for which g{ta) < 1 
in order for (l26l ) to hold. Similarly, if for all n large t' G (0, A™"^), then it is necessary and sufficient to find a 
tb G {0,t') for which gn{tb) < 1 in order for ^ to hold. Notice if c(l - H{0)) > 1 then gn{t) > 1 for all 
t < A5f™ and all n large. 

Let for d.f. G with bounded support, Xq^^ denote the largest number in Sc- If there is a r > A^'^^ for 
which ^(t) = cJ jj^-j^dH{t) = 1, and if limsup„A™'^^ < r, then r can be used as a threshold for t' G 

(limsup„ A™'^'^, 00). Indeed, by the dominated convergence theorem, lim„_^oo 9n(t') = g{t')- Therefore, if t' > r, 
conditions (b) and (c) of Theorem 16.31 hold, with ta = r, and tb any arbitrarily large number. 
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On the other hand, suppose A - where £ remains bounded, are the eigenvalues of T„ approaching 

the interval (limsup^ ^n^^^i 't]- Then by Theorem I6.4[ for any e > with probability one, none of Af " , . . . , \f\(^_i 
can remain in (A™*^^ + e, oo) with for all n large. 

Also, since the largest i„ + £ — 1 eigenvalues of T„ must be o{n) (otherwise, H would have additional mass on 
[Ag^, oo)), A^", . . . , A^"_(_^_-^ must all converge a.s. to A™''''. 

Similar results can be obtained for the interval to the left of Sp 

As in Theorem 16 . 1 1 Tonelli ' s Theorem can easily be applied to establish equivalent results when T„'s are random 
and independent of X. 



E. The eigenvalues of the multivariate F matrix 

Let Yij be another collection of i.i.d. random variables (not necessarily having the same distribution as the 
Xjj's), with EYii = 0, E|Yii| = 1, E|Yii|'^ < oo, and independent of the Xj^'s. 

We form the n x iV matrix Y„ = {Yij), i = l,2,...,n, j = 1,2,...,N with N = N{n), n < N, and 
= n/A^ — > ci G (0, 1) as n — > oo. 

Let now T„ = ((1/A^)Y„Y*)~^, whenever the inverse exists. 

From Bai and Yin's work [39] we know that with probability 1, for all n large, T„ exists with A^" (1 — y^)"^. 
Whenever X^n^^^^"^" = define T„ to be I. 

The matrix T„(1/A)X„X*, typically called a multivariate F matrix, has the same eigenvalues as B„. Its limiting 
e.d.f. has density on (0, oo) given by 

f ( X {\-cv)sJ{x-hx){h2-x) 

^'='-(^) = 2vr.(xci + c) f'i<-<^2, 

where ^ ^ 

^ 1^ 1- Vl-(l-c)(l-ci) ^ ^ ^ 1^ 1 + Vl-(l-c)(l-ci) ^ 

When c G (0, 1], there is no mass at 0, whereas for c > 1 F has mass (1 — (1/c)) at [14]. 
We are interested in the effect on spikes on the right side of the support of the 

Because of the corollary to Theorem 16. 1[ we know Af " — > 62 a.s. as n — > 00. We proceed in computing the 
function ^ 



= ^ / (A-t)2 ^^(^^- 



We will see that it is unnecessary to compute the limiting e.d.f of T„. It suffices to know the limiting Stieltjes 
transform of F(VA^)Ky„- 

Let Hx denote the hmiting e.d.f. of F^^/^)^"^". We have 



9 3j , ^ 



x={l/t) 



We use (1211) to find mpal,I^^^ 



11 9 . 

hci <^ zm^ + (z + 1 - ci)m + 1 = 0. 

m 1 + m 



-z-l + ci± J{z + l-ci)^ -Az 

<^ m = 

2z 

(the sign depending on with branch of the square root is taken). 

_ _^ _ 1 + ± ^(^ _ (1 _ ^)^)(z - (1 + ^)2) 

2z 
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From the identity in ( [201 ) we find tiiat 



niH^ {z) 



2ciz 

As mentioned earlier the support of Hi is [(1 — -^/ci)^, (1 + ^/cl)'^]. We need g{t) for t > (1 — y^)~^, so we 
need muiix) for x G (0, (1 — y/ci)^). 

Since G 5^^, m//^(0) exists and is real, which dictates what sign is taken on (0, (1 — y/ci)^). We find that, 
on this interval 

mH^{x) = , (29) 

IClX 

and using the fact that the discriminant equals x"^ — 2x(l + ci) + (1 — ci)^, 

d 1 / X(l + Ci) - (1-Ci)2 ^ 

— m^^,(x) = -— -2 (l-ci) + 



We therefore find that for t > (1 - ^/cl)"^ 

5W = 7^ -(l-ci) + 



2ci V ^(l-t(l-y5r)2)(l-t(l + ^)2)y 

We see that the equation g{t) = 1 leads to the following quadratic equation in t: 

(1 — ci)'^at^ — 2(1 + ci)at + a — = 0, where a = ci + c — cci, 

giving us 

^ _ (1 + ci)a + V(l + ci)^a2 - (1 - ci)2a(a - c^) 



(l-ci)2a 

The positive sign in front of the square root being correct due to 

(1 + ci) _ (1 + ci) ^ 1 



(1-Ci)2 (1-^)2(1 + ^^)2 (1-^)2- 

Reducing further we find the threshold, r, to be 



(1 + ci)a + ^y/4a - ci + (1 - ci)2c2 ^ (1 + cQa + V^(2ci + c(l - d)) ^^^^ 
(1 — ci)2a (1 — ci)2a 

We now compute the right hand side of (l26l ). We have for t' > t 



t'(2ci + c(l - ci)) - c - c^(l - t'(l - V^)2)(l - t'{l + ^) 



A(t'). (31) 



2ci 

A straightforward (but tedious) calculation will yield A(r) = 62- 

Using the results from the previous section, we have proved the following: 
Theorem 6.5: Assume in addition to the assumptions in Theorem 16. II on the Xjj 

(a) the T„, possibly random, are independent of the Xjj, with F^" -^n H, a.s. as n ^ 00, H being the limiting 
e.d.f. of f((^/^)^"^")"\ defined above. 

(b) Almost surely, there are £ (remaining finite for each realization) eigenvalues of T„ converging to nonrandom 
t' > (1 — y/ci)~'^, as n ^ 00. Denote by iJ„ the e.d.f. of the n — £ other eigenvalues of T„. 

(c) With A™^^ defined to be the largest number in the support of Hn, with probability one, limsup„ A™^^ < r 
the threshold defined in (l30l ). 

Suppose Xj" , . . . , Xj"^^_^ are the eigenvalues stated in (b) of Theorem 16.31 Then, with the function A(-) defined 
in (|3TI ). with probability one 

lim Xf" = ■ ■ ■ = lim A^", „ 1 = < ^ , 
r^^o^ ^^o^ \ 62, if G (limsup„AJf^^,r]. 

Note: From Theorem 16. 3[ when t' > t the result can allow I = o{n). 
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